add local-telemetry stack for investigating server performance #16483
Conversation
zzstoatzz
force-pushed
the
otel-oss
branch
3 times, most recently
from
8cec80b to
d5c7583
Compare
CodSpeed Performance ReportMerging #16483 will not alter performanceComparing Summary
|
zzstoatzz
requested review from
aaazzam,
chrisguidry,
cicdw,
desertaxle and
discdiver
as code owners
src/prefect/main.py
Outdated
| if not os.environ.get("PREFECT__ENABLE_OSS_TELEMETRY"): | ||
| import prefect.telemetry.bootstrap | ||
|
|
||
| prefect.telemetry.bootstrap.setup_telemetry() | ||
| prefect.telemetry.bootstrap.setup_telemetry() |
There was a problem hiding this comment.
Could the short circuit that you need be moved to setup_telemetry or prefect.telemetry.bootstrap instead of being in prefect.main?
There was a problem hiding this comment.
hmm i had intentionally guarded the imports as well but let me see if i can move it inside
There was a problem hiding this comment.
I was thinking that the env var you're checking for will be unset in the large majority of cases, so it'd be better to be confined to that module. What's the reason for not importing that module if this environment variable is set?
There was a problem hiding this comment.
that makes sense, i was just under the impression that the imports themselves for the existing OTEL setup would interfere with the new local telemetry. but i haven't checked yet so that might not be true - will update
There was a problem hiding this comment.
@desertaxle because im targeting create_app directly
PREFECT_API_URL=http://localhost:4200/api \
OTEL_SERVICE_NAME=prefect-server \
OTEL_TRACES_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
OTEL_LOG_LEVEL=debug \
PYTHONPATH=/Users/nate/github.com/prefecthq/prefect/src \
opentelemetry-instrument \
uvicorn \
--app-dir /Users/nate/github.com/prefecthq/prefect/src \
--factory prefect.server.api.server:create_app \
--host 127.0.0.1 \
--port 4200 \
--timeout-keep-alive 5
I actually don't need this short circuit at all, so I removed it
| prefect --no-prompt work-pool create local --type process --overwrite | ||
|
|
||
| prefect --no-prompt deploy --all --prefect-file load_testing/prefect.yaml |
There was a problem hiding this comment.
populate the server
load_testing/run-server.sh
Outdated
| PYTHONPATH=/Users/nate/github.com/prefecthq/prefect/src \ | ||
| opentelemetry-instrument \ | ||
| uvicorn \ | ||
| --app-dir /Users/nate/github.com/prefecthq/prefect/src \ |
There was a problem hiding this comment.
Might need to make the app-dir configurable or use a relative path.
|
good catches @desertaxle - made those changes |
Co-authored-by: Alex Streed <[email protected]>
related to #16299
this doesn't make any performances changes yet, instead it sets up a local telemetry stack we can use to explore OTEL data in Jaeger, which should make it easier to support / confirm suspected performance issues related to that issue